Fine grained network management to edge device features

ABSTRACT

Network management systems and methods are provided. A system is provided that includes an event database that provides a mapping of multimodal sensor data to events of devices coupled to a network, such as operational or behavioral events. A network management engine obtains a set of multimodal sensor data relating to a device, which may include sensor measurement or output data relating to each of multiple device operation or behavior parameters. The engine determines, utilizing the mapping, a match of the set of multimodal sensor data to a specific event associated with the device. Based at least in part on the determined match, the engine causes generation or updating of event data associated with the specific event, the data being accessible by network management software of the network management system.

BACKGROUND

This disclosure relates generally to network management and managementof devices over a network. More than ever, a variety of devices in anynumber of disparate applications and uses, from components of cars tomedical monitoring equipment, are coupled to, and exchange informationover, networks. Such devices may include instrumentation allowing, forexample, collection and communication, through the network, ofinformation related to operation of the device or its components.Furthermore, remote communication to the device, or between devices, mayalso be utilized, for example, in providing updates, remote monitoringor control, coordination, or for many other reasons and uses.

For example, in what has come to be termed the “Internet of Things”,devices such as vehicles, appliances, and any number of other types ofitems, may be instrumented with sensors, electronics, software,interfaces, etc., allowing collection and exchange of data over theInternet and other networks.

There are known technologies, in various fields, that include collectionof data, including sensor data, for use in identifying and alerting orresponding to conditions relating to devices or other entities.

Some technologies attempt to address problems related to location andactivity identification. For example, U.S. Pat. No. 9,560,094, entitled,“System and method for identifying and analyzing personal context of auser” includes the use of various types of sensor data to deduce, withcertain degrees of confidence, context information about a user, such aslocation, identity and activity. The sensor data can include data fromexternal sensors as well as data from sensors embedded in a mobilecommunication device.

Other technologies attempt to address audio or visual problems, such asconstructing or directing views, etc. For example, U.S. Pat. No.8,754,925, entitled, “Audio source locator and tracker, a method ofdirecting a camera to view an audio source and a video conferencingterminal” uses data from at least two different types of sensors tolocate an audio source, such as in a video conferencing context.

Other technologies are generally related to batteries, such astechnologies directed to conserving or improving battery life. Forexample, U.S. Pat. No. 9,929,772, entitled, “Low power, high resolutionautomated meter reading and analytics”, is directed to problemsincluding improving battery life performance in end devices in resourcemonitoring systems, including sensor interfaces in end devicesconfigured to communicate with multiple types of sensors. Such sensorscan include 3-wire Automatic Meter Reading (AMR) sensors, Hall effectsensors, reed switch sensors, pulses sensors, and magneto-resistivesensors. Generally, different duty cycles may be utilized in the enddevices, relating to power usage associated with sensor data collectionand usage, to conserve battery life associated with the end device.

Sensor technologies have been developed that are directed to carbatteries, in particular. For example, U.S. Pat. No. 9,640,845,entitled, “Temperature-raising device and temperature-raising method forin-car battery”, discusses use of a temperature sensor installed aroundan in-car battery, to obtain car battery temperature data in connectionwith use of a car battery heater system.

Various sensor and alert technologies exist in the context of vehicles.For example, in U.S. Pat. No. 9,919,646, entitled, “Sound, temperatureand motion alarm for vehicle occupants and pets”, various types ofdetectors and sensors are used in connection with a vehicle temperaturealarm system. In general, when certain detectors or sensors pick upcertain conditions, other detectors may be turned on as a result. Thiscan ultimately lead to a motion detector being turned on, which maysound an alarm upon sensing of motion. Other sensor technologies arespecifically associated with detecting drunk driving. For example, U.S.Pat. No. 8,941,501 is directed to a system that includes preventingstart of the vehicle if an alcohol sensor detects a high enough alcoholcontent by volume in the breath of the would-be driver.

Sensor system technologies also exist for use in medical monitoring. Forexample, U.S. Pat. No. 9,907,503, entitled, “Sensor systems and methodsof using the same”, is directed to utilization of a selectivelypositionable sensor system, to sense one or more analytes in a bodilyfluid, for use in controlling fluid administration to a patient.

SUMMARY

The above-referenced known technologies generally have limitations inareas including, for example, practical utility, overall sophisticationand granularity, and overall accuracy and precision including withrespect to determination of thresholds or conditions, and many of thesetechnologies have not been adapted to use, or at least in some cases notadapted to fully leverage, the advantages of network connectivity. Thereare potential benefits that have not yet been realized in scenarios andapplications including exchange of device related information over anetwork, such as, for example, “Internet of Things” applications. Manyunrealized benefits can be realized to the extent that more useful,accurate, or very granular device-related information can be efficientlycollected from devices and that information can be effectively put touse in managing edge devices. Such benefits include those in areas suchas device operation, efficiency, monitoring, alerting, visibility,control, automation, optimization, determination or recognition ofevents, and others.

In some embodiments of the invention, a device is instrumented, whichcan include equipped with or coupled to sensors, hardware, software,interfaces, etc., to collect multimodal sensor data (i.e., one or moredata types representing one or more sensor modalities), such as mayrelate to multiple operational parameters of the device or a componentof the device, and to communicate the data over a network. The data may,for example, relate to multiple operational parameters of a device,device set or group, or a component of a larger device or item, such asa battery of a vehicle or device. The data may be transmitted, includingwirelessly, over the Internet and other networks. In some cases, thedata may be processed or filtered at the device or otherwise prior tobeing transmitted over the network, where it may be further processedremotely, etc.

In some embodiments, remotely from the device, an event database ismaintained. The event database may associate, or map, sets of multimodalsensor data to associated events (which can include, among other things,occurrence or continued occurrence of particular statuses, states,conditions, etc.), such as operational or behavioral events related tothe device. In some embodiments, behavioral events can relate tobehavior of people or animals as sensed or otherwise digitally observed.In some such cases, the device or devices involved may gather datarelating to the person, for example, and an event associated with thedevice could mean, for instance, a behavioral event associated with theperson whose data is being collected by the device. In some embodiments,the event database, which may be structured as a Management InformationBase (MIB), for example, may effectively reflect or model, structurally,organizationally, verbally, hierarchically or ontologically, aspects ofa subject, such as operational aspects of a device, component set, orcomponent, or even of a larger structural entity such as a roadway orbuilding, or behavioral aspects of a person or an animal, group,environment, ecosystem, etc.

In some embodiments, mapping includes ranges or thresholds for each ofmultiple sensed parameters, such as measurement data or sensor outputdata, which ranges or thresholds must be met in order for a match to bedetermined. In some embodiments, more complex mapping techniques ormodels may be used, such as may use functions, variables, graphingtechniques, vector modelling, or other techniques.

In some embodiments, remotely from the device, a network managementengine, coupled to the event database, such as may be implemented usingone or more servers, obtains the set of multimodal sensor data. Thenetwork management engine may further match the set of multimodal sensordata to a particular event associated with a device or other entity(e.g., person, animal, environment, etc.), which event may be determinedor deemed to have occurred or be occurring, or to be likely to haveoccurred or be occurring (e.g., a vehicle battery overheating, or someother component experiencing some condition).

In some embodiments, based at least in part on the determined match tothe particular event associated with the device, the network managementengine generates or updates event data associated with the event, whichevent data is accessible by network management software.

In some embodiments, after the event data is detected, actions may betaken, such as determining, and communicating over a network, system oruser alerting, device operation modification or optimizationinstructions or commands, etc.

In some embodiments, events may relate to human or animal behavioralevents as determined from multimodal sensor data. For example, in ahealth care or medical context, monitored patient medical parameterinformation can be used to assess or determine a patient-related healthevent, or, in a consumer related context, shoplifting or fraud may bedetermined, etc. Many other applications and contexts are possible.

In some embodiments, existing network infrastructure, protocol,standards, etc., may be leveraged in data collection, processing andcommunication. Furthermore, machine learning techniques may be utilizedto repeatedly or iteratively modify and optimize the mapping of theevent database. Sensed parameter data and related results data, as wellas mapping and other data, may be used as training data for the machinelearning techniques or models.

Generally, techniques according to embodiments of the invention canprovide advantages in many ways and in disparate applications in networkmanagement, and management of devices or entities coupled to orassociated with a network.

Various other aspects of the inventive subject matter will become moreapparent from the following specification, along with the accompanyingdrawings in which like numerals represent like components.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a networked system in accordance with an embodimentof the invention, including a network management engine, event databaseand devices.

FIG. 2 illustrates a block diagram in accordance with an embodiment ofthe invention, including collection and storage multimodal sensor datarelating to a vehicle battery.

FIG. 3 illustrates a block diagram in accordance with an embodiment ofthe invention, illustrating event database mapping and use of eventdata.

FIG. 4 illustrates a block diagram in accordance with an embodiment ofthe invention, illustrating components of a network management engine.

FIG. 5 illustrates a block diagram in accordance with an embodiment ofthe invention, illustrating roles of a network management engine and anevent database.

FIG. 6 illustrates a flow diagram of a method in accordance with anembodiment of the invention, including generation or updating of eventdata upon a determined match.

FIG. 7 illustrates a flow diagram of a method in accordance with anembodiment of the invention, including monitoring for new multimodalsensor data.

FIG. 8 illustrates a flow diagram of a method in accordance with anembodiment of the invention, including utilizing machine learning toupdate an event database.

FIG. 9 illustrates a block diagram of a distributed computer system thatcan implement one or more aspects of a network management system ormethod according to an embodiment of the invention.

FIG. 10 illustrates a block diagram of an electronic device that canimplement one or more aspects of a network management system or methodaccording to one embodiment of the invention.

While the invention is described with reference to the above drawings,the drawings are intended to be illustrative, and other embodiments areconsistent with the spirit, and within the scope, of the invention.

DETAILED DESCRIPTION

Herein, “sensor data” can broadly include, for example, output ormeasurement data from sensors, or data derived therefrom or basedthereon, such as after being processed, filtered or modified. Typicallysensor data is in a digital format or converted to a digital formatunderstandable by a computer. For example, an analog device (e.g.,thermostat, etc.) could generate an analog signal that is converted to adigital value via an analog to digital converter (ADC). Further, the rawdigital value can be further converted or formatted for consumption byhumans or computers; converted to XML or JSON format for example.

Herein, “behavior” and “behavioral” can broadly include, for example,the manner of function, operation or conduct, of an entity; a device,people, person, animals, groups, or other entity capable of respondingto stimuli.

Herein, “device” can broadly include one or more aspects, components orelements of a device; typically a computing device having at least onenon-transitory computer readable memory (e.g., RAM, flash, HDD, SSD,etc.) and at least one processor.

Herein, a set of “multimodal sensor data” can broadly include, forexample, a set of data that includes sensor data relating to each ofmultiple parameters including, in some cases, data sensed usingdifferent modalities.

Herein, an “edge device” can broadly include, for example, a device atthe conceptual “edge” of a network, such as end devices or user devices,among other things, including devices in Internet of Thingsenvironments, for example. Furthermore, an “edge device” can include,for example, a device which provides an entry point into a network, suchas routers, routing switches, integrated access devices (IADs),multiplexers, metropolitan area network (MAN) and wide area network(WAN) access devices, and others.

Herein, an “event” can broadly include, for example, particularoccurrences, happenings, occurrences, activities, or actions, which caninclude occurrence or continued occurrence of particular statuses,states, conditions, etc., and can also include network, software orcomputer-related events, etc.

FIG. 1 illustrates a networked system 100 in accordance with anembodiment of the invention. A number of devices 106 and networkmanagement engine 102 are coupled to a network 106. The networkmanagement engine 102 is coupled to event database 104. Each of thedevices 106 includes multiple components 108, and each includes or iscoupled to network management related device instrumentation 110, whichmay broadly include, for example, hardware, software, applications,CPUs, data storage, sensors, interfaces, wired interfaces, wirelessinterfaces, or other elements.

FIG. 2 illustrates a block diagram 200 in accordance with an embodimentof the invention, including collection and storage multimodal sensordata relating to a device, specifically, a battery 202 of a vehicle 204.In example scenario illustrated, battery 202, which includes variousbattery components 210, includes or is coupled to instrumentation 206for uses that may include collecting, processing, filtering and sendingmultimodal sensor data for storage in an event database 208.

FIG. 3 illustrates a block diagram 300 in accordance with an embodimentof the invention, illustrating event database mapping and use of eventdata. As depicted, many data sets of several different types of datasets 302, 304, 306 may be sent to, and stored in, operational orbehavioral event database 308. The data can include multimodal sensordata 302, which can include, for example, data derived from sensor data,including after being processed, filtered or modified. The data canfurther include result or outcome data 304, which can, for example,relate to results or outcomes of determined events associated withmultimodal sensor data, or results or outcomes of actions taken based onsuch events, or based on detected event data that was generated orupdated based on such events. The data can further include other datauseful in mapping to operational or behavioral events, which caninclude, for example, being useful in recognizing events, definingevents or criteria, identifying events, defining, updating or modifyingmapping, etc.

One or more machine learning models 310 may be used in connection withthe database 308, such as to update, modify or better optimize thedatabase 308, which may be done iteratively and frequently. Various datamay be used as input or training data to train, optimize or modify themachine learning model(s) themselves. For example, any of data 302, 304and 306 may be so used, or other data, such as from the database 308,including mapping related data. As illustrated, mapping 312 of thedatabase 308 can include, for example, a set of specific definitions orrequired criteria of specific events, such as operational or behavioralevents. For example, definitions can be or include specifications orrequirements in terms of multimodal sensor data, thresholds (which mayrelate to each of multiple parameters, measurements, or outputs, forexample), ranges or limits, criteria relating to other data such asresult or outcome data, or complex definitions, such as multivariablefunctions or equations that may include multimodal sensor data withinthe functions, such as by being variables within the functions.

As illustrated, network management engine 314 may generate or updateevent data 316, such as when it is determined that a set of multimodalsensor data matches the definition of specified event, which mayindicate that the event is determined or deemed to have occurred or tobe occurring. Furthermore, as illustrated, network management software318 (which, in various embodiments, may or may not be part of thenetwork management engine 314 itself) may, for example, detect, ormonitor for and detect, event data 316, and may cause, facilitate orhelp facilitate the taking of some action 320 based at least in part onthe detected event data 316, such as, for example, an operational orbehavioral modification or optimization, or issuing of instructions orcommands therefor, or alerting, whether, for example, intra-system oralerting or alerting for display to and use of a user, etc.

FIG. 4 illustrates a block diagram 400 in accordance with an embodimentof the invention, illustrating components of a network management engine406, which includes device network management instrumentation elements404 and remote elements 408. Device network management instrumentationelements 404 may, for example, relate to collection, local storage,management, filtering and processing of multimodal sensor data prior towireless transmission to remote elements of the network managementengine or other devices. Remote elements 408 may, for example, beimplemented by one or more servers after multimodal sensor data (whichmay be filtered or processed data) is received after being wirelesslytransmitted from the device.

FIG. 5 illustrates a block diagram 500 in accordance with an embodimentof the invention, illustrating roles of a network management engine 502and an event database 504.

In some embodiments, the event database 504 stores multimodal sensordata relating to events, such as operational or behavioral events,relating to edge devices, such as batteries, vehicle batteries or othercomponents, manufacturing devices, medical/medical monitoring relateddevices, consumer/consumer behavior monitoring related devices, etc. Itmay also store logs of such data, or such data over time, such as byusing blockchains or other distributed ledger or data structures. It mayfurther store additional data such as outcome data relating to resultsor outcomes associated with behavioral events. Furthermore, it mayprovide a mapping of sets of multimodal sensor data to events, such asmay relate, for example, to device operation or device componentoperation or status conditions. In some cases, multiple parametermeasurements may be included in a mapping to a particular behavioral oroperational event. For example, in some cases, each of multipleparameter measurements must fall within particular ranges or meetparticular thresholds, or satisfy more complex criteria that may includeone or more functions, for example, in order for a particular event tobe reflected by the mapping. In some embodiments, the Event databasescan be or include elements of a MIB, and may use such protocols orstandards as Simple Network Management Protocol (SNMP) and OpenNMS (seeURL www.opennms.com). It may also be structured hierarchically andontologically in connection with categories and terms relating to thebehaviors.

In some embodiments, the network management engine 502 obtainsmultimodal sensor data relating to edge devices. Furthermore, it may,utilizing the event database 504, determine a matching between aparticular set of multimodal sensor data and a particular behavioralevent, which may indicate that the behavioral event has occurred or iscontinuing to occur. The network management engine 502 may also generateor update event data associated with the event, which may be accessed bynetwork management software. In some embodiments, it may monitor for anddetect generated or updated event data. Furthermore, it may causeactions to be taken upon such detection, such as generation or displayof alerts, or causing modification of operation of the relevant deviceor component, such as to optimize operation thereof. Still further, itmay utilize one or more machine learning techniques or models inconnection with the event database 504, such as to detect event andoutcome patterns, or to modify or optimize mapping relating tomultimodal sensor data and events.

FIG. 6 illustrates a flow diagram of a method 600 in accordance with anembodiment of the invention, including generation or updating of eventdata upon a determined match. At step 602, a network management engineobtains multimodal sensor data relating to an edge device.

At step 606, the network management engine queries whether a particularset of multimodal sensor data matches to a specified event. If no matchis determined, then, at step 608, no event data is generated or updated.However, if a match is determined, then, at step 610, the networkmanagement engine causes generation or updating of event data associatedwith the specified event, which data is accessible by network managementsoftware.

FIG. 7 illustrates a flow diagram of a method 700 in accordance with anembodiment of the invention, including monitoring for new multimodalsensor data. At step 702, a network management engine obtains multimodalsensor data relating to an edge device. At step 704, the networkmanagement engine monitors for newly obtained multimodal sensor datamatching an event.

At step 706, the network management engine then queries whether aparticular set of multimodal sensor data matches to a specified event.If no match is detected, the method 700 proceeds back to step 704, atwhich the network management engine again monitors, or continues tomonitor, for newly obtained multimodal sensor information. However, if amatch is detected, the method proceeds to step 708, at which the networkmanagement engine causes generation or updating of event data associatedwith the event, which data is accessible by network management software.

FIG. 8 illustrates a flow diagram of a method 800 in accordance with anembodiment of the invention, including utilizing machine learning toupdate an event database 816. At step 802, a network management engine812 obtains multimodal sensor data or other data 810 associated with anedge device.

At step 804, the Network management engine iteratively utilizes, astraining data for a machine learning model(s) 814, newly obtained dataincluding obtained multimodal sensor data, or sensor data logs, as wellas obtained outcome data relating to results of events or actions takenbased on events.

At step 806, the machine learning model(s) 814 are used in updating andmodifying event database 816, including mapping of multimodal sensordata to events associated with edge devices.

At step 808, the updated and modified event database 816 is used infuture matching determinations of sets of multimodal sensor data tobehavioral events, and may be used in other ways as well. Thereafter,the method 800 returns to step 802, where the network management engine812 obtains new multimodal sensor data and other data relating to, orassociated with, an edge device.

In some embodiments of the invention, existing network management tools,structures, protocols, standards, etc. are used in creative ways. Forexample, such tools may be used to characterize, at increasingly finelevels of detail or granularity, the workings of devices, such as edgedevices in Internet of Things scenarios, as well as sets of components,individual components, etc. Devices can be instrumented to allow, amongother things, this collection and organization of useful data. Forexample, in some embodiments, the OpenNMS network monitoring and networkmanagement platform may be used in obtaining such fine-grained deviceoperational data. Furthermore, databases according to some embodimentsmay utilize or be a MIB, and utilize protocols associated with suchdatabases. This particularly may include SNMP. MIB and SNMP can be usedto effectively manage and organize data about edge devices as wellinterfacing devices. MIB and SNMP can also be used effectively inembodiments of the invention to structure, organize and utilize devicerelated data in modifying device operation or behavior, such as upondetection of operational or behavioral events based on multimodal sensordata, for instance.

Furthermore, in some embodiments, a MIB and mapping utilized can bestructured hierarchically and ontologically, such as a MIB of behavioralor operational events, or events in particular contexts, etc. This couldinclude, in effect, a MIB of any number of devices in automotive,manufacturing or other industries, such as, for example, zinc-airbattery operation, including a hierarchy that may reflect larger andmore fine-grained components and subcomponents, and relationshipsbetween them, etc. Another very different example could be a MIB of, forexample, consumer behavior, which could higher and more fine-grainedactions or conduct, such as actions relating to lead-up to purchasingand actual purchasing, etc. Other examples could include MIBs of smallercomponents, or even MIBs are larger scale structures, like an entirevehicle or roadway, etc. In some embodiments, MIBs, as event databases,could even be used to reflect aspects of entities or individualsthemselves, such as patients, or even groups, social networks,organizations, infrastructures. For example, in some embodiments, a MIB,or associated mapping can be organized such that data associated withmultiple parameters may be utilized to determine whether specified,context-specific events have occurred or are occurring.

For example, with a battery, such as a lithium ion battery, zinc-airbattery, etc., multiple technical battery parameters or attributes maybe represented in the MIB and associated with a definition of an event,such as overheated status or lower power status occurring. Furthermore,for example, battery or power management device components could besubject to instrumentation and drill-down on a fine-grain andmulticomponent basis. The cumulative, structured data can be reflectedin the structure of the MIB and mapping to particular operationalevents.

Some embodiments of the invention can be utilized in connection withvehicle batteries, such as car batteries (even potentially including,for example, now or in the future, metal-air or zinc-air batteries orother energy storage devices). In some embodiments, multimodal sensordata can be collected and utilized in connection with car batteries, andcomponents thereof. Car battery sensors and interfaces may be provided,integrated or incorporated into the battery, mounted or installed tomeasure parameters relating to the battery or particular components orsubcomponents thereof. Such parameters that may be sensed, or may bedirected indirectly detected, calculated or deduced from sensed data,may include, for example, operating temperature, temperature gradientsor differences, voltage, nominal cell voltage, current, charge,discharge, transient response, internal resistance, internal resistancecathode area, air availability, air distribution, oxygen, porosity,catalytic value of the cathode surface, capacity, power, specific power,energy, age, or other electrical, chemical or mechanical parameters. Themultimodal sensor data can then be used to determine whether operationalevents, such as overheating or low power, for example, or conditions orthresholds related to particular parameters, have occurred. In someembodiments, sensors and interfaces may be incorporated into a modulethat can be mounted onto the battery.

Battery-related operational events may be determined based on eventdefinitions or thresholds stored in an event database, utilizingmultimodal sensor data, and potentially other data, such as data fromcontrol units within the car electrical system. Based on determinedevent data, various actions may be initiated or taken, such as alertingor car or battery operational commands or actions, such as may involvevarious aspects of battery operation, including cycling, charging, orheating of the battery.

The following provides an example of one embodiment of the invention, inconnection with a zinc-air battery. These batteries are generallyconsidered safer than various other types of batteries. Moreover, sincezinc-air batteries use atmospheric oxygen as a cathode reactant, asopposed to using a cell-contained chemical reactant, a zinc-air batterycell can provide a very high energy density, thus allowing the zinc-airbattery to be lightweight, compact and high capacity. Thesecharacteristics have led to increasing use of zinc air batteries inapplications including, for example, providing power to electronic andtelecommunications devices, medical monitoring devices, and evenelectric vehicles.

Performance of zinc-air batteries is affected by factors, or parameters,notably including temperature, ambient relative humidity, dischargerate, and the percentage of capacity remaining in the battery cell.Moreover, performance is affected by a complex interplay between thesefactors.

In particular, voltage provided by a zinc-air battery is affected byeach of these parameters, and their combinations. For example, azinc-air battery may typically operate well at a temperature of 0-50°C., and an ambient relative humidity of between 40-60%, but performancewill also be influenced by parameters including discharge rate andpercentage of capacity remaining in the battery cell. Maintaining anoptimal (or sufficient) provided voltage output, or voltage outputrange, may depend not only on each of these parameters being withincertain ranges, but also on their values or value ranges relative toeach other. As such, values for a particular parameter, such astemperature, that threaten optimal voltage output can vary depending onthe values of the other parameters. As one specific example of this,while lower temperature generally leads to lower voltage output, so doeslower remaining capacity. So, for example, in a hypothetical case,keeping other parameters constant, at 100% remaining capacity, atemperature of 10° C. or lower may cause voltage to drop below theoptimal value or range, whereas, at 25% remaining capacity, atemperature of 5° C. or lower may be required to cause voltage to dropbelow the optimal value or range.

In spite of this complex interplay between the four mentioned parametersas affecting output voltage of a zinc-air battery cell, therelationships of them at various value ranges can be experimentallydetermined, for a particular type of zinc-air battery cell. This, inturn, can be used, for example, to determine combinations of valueranges for each of the four parameters that are considered to place adefined optimal (or sufficient) voltage output in jeopardy. As such, forexample, even if voltage can be sensed or measured directly, and fallswithin the optimal range, it may be noteworthy and useful to detect whencombinations of the four previously mentioned parameters place optimalvoltage output in jeopardy, such that if one or several of them varyeven slightly (such as by a certain small percentage) in a particulardirection (such as, for example, a 1° C. temperature drop), optimalvoltage output may fail. What combinations of the parameters suffice toconstitute “jeopardy” in this regard can defined, for example, based onone or several of the parameters being extremely close, such as within asmall, defined percentage, of that which would, in combination with aparticular set of other parameter values, lead to output voltagedropping below the optimal (or sufficient) value or range.

Given all of the above, for a particular zinc-air battery, in oneembodiment of the invention, the zinc air battery may be instrumentedwith, or to include, sensors to allow frequent or constant multimodalsensor data collection and measurement of each of these parameters overtime. An optimal (or sufficient) “output voltage jeopardy” operationalor behavioral event may be defined to include various combinations ofsets of value ranges for each of the four parameters (it is noted that,in other embodiments, other or different parameters, potentiallydirectly including voltage, could also be included or accounted for). Itis notable that, at a given time, even if voltage is within the optimalrange, an output voltage jeopardy event may still occur. Moreover, giventhe interplay between the parameters, even if none of the individualparameters alone would indicate jeopardy, the combination of them mightstill come sufficiently close to causing output voltage to drop belowthe optimal range, and thus might still trigger an output voltagejeopardy event. As such, this demonstrates an example of how someembodiments of the invention allow, among other things, recognition ofsignificant events that might otherwise go unrecognized or undetected.

Continuing with description of the above example, a MIB could bedesigned to include definitions and mapping of various operationaland/or behavioral events relating to the particular type of zinc-airbattery. Operation or behavior of the battery cell can be monitored.When combinations of values of particular parameters, as measured usingmultimodal sensor data, fall within certain defined ranges, they can bematched to a defined event, such as, in the above-described example, anoptimal (or sufficient) output voltage jeopardy event. Based on thisdetermined match, event data can be generated or updated. This caninclude, for example, utilizing an OpenNMS platform in receiving ordetecting event data, generating notifications or alarms, issuingcommands, etc. Continuing with the above example, once an optimal (orsufficient) output voltage jeopardy event is detected, notifications canprovided, steps taken, or operational commands issued, to prevent theparameters from further changing such as to cause voltage to drop belowthe optimal (or sufficient) range (or value). There are many examples ofpossible such steps or operational commands. Some could include, forexample, a vehicle driver being notified or alerted to the situation,and to ensure that driving does not continue into a lower ambienttemperature range that would cause drop in voltage. As another example,an operational command could be (including wirelessly) provided to aheating system or element that may be installed with or near thezinc-air battery to cause the temperature not to drop such as to causedrop in voltage, etc.

Further to the above, Table 1, below, provides an illustrative,hypothetical example of different sets, or combinations, of parametervalues or value ranges that may constitute or set forth, in whole or inpart, definitions of particular varieties or subcategories of outputvoltage jeopardy events, for a particular zinc-air battery. In thishypothetical example, each row defines a particular event, of OutputVoltage Jeopardy Events A-D. Definitions of events, including theseevents, may be, for example, reflected in a MIB. As shown, for a givenrow, if all of the four parameters fall within the specified ranges,then the event definition is met, event data is generated or updatedaccordingly, Open NMS may be used in detection and generatingnotifications, etc. Flows or algorithms can be utilized in determiningif an event is triggered. As a simple example, For Output VoltageJeopardy Event A, such a flow could include a conditional such as:

-   -   IF % Remaining Capacity is in the range of 25.1 to 100 AND        Discharge Rate in Hours=290-310 AND temperature in ° C.=0-4.99        AND Relative Humidity in %=40-60 THEN Output Voltage Jeopardy        Event A is triggered.

TABLE 1 Parameter Value Range Combinations associated with DefinedEvents, as can be reflected in an associated MIB EVENT (triggered if allfour parameters in the row fall within the specified Relative ranges ormatch the % of Capacity Discharge Rate, Temperature humidity specifiedvalue) Remaining in Hours ° C. In % Output Voltage 25.1-100 290-310 0-4.99 40-60 Jeopardy Event A Output Voltage 25.1-100 140-160  0-4.9935-39.9 or Jeopardy Event 60.1-80 B Output Voltage 0.1-25 290-3105.0-10.0 40-60 Jeopardy Event C Output Voltage 0.1-25 140-160 5.0-10.035-39.9 or Jeopardy Event 60.1-80 D

Some embodiments of the invention are used in connection with drunkdriving. In some embodiments, multimodal sensor data is used todetermine if drunk driving is or may be occurring, so that action can betaken accordingly, such as alerting (of the potential driver, orremotely, such to the police or some other entity or agency), ordisabling of the vehicle. A sensor for detecting alcohol by volume in abreath of the potential driver can be one parameter that is measured,but it could be used on combination with a number of others.

In some embodiments, in-car and exterior video, camera, 3d or depthcamera, and audio feeds can be utilized and analyzed as part of themultimodal sensor data, including of the car interior and exterior,surroundings, and the driver himself, which may incorporated into thedefinition or threshold of a drunk driving related event. In someembodiments, breath alcohol level could be one factor. The driver'sidentity or other personal characteristics could be utilized. Light andmotion sensors within the car could also be utilized. Sensor systemscould be positioned and deployed optimally for these purposes. Variousother operational parameters could also be sensed and utilized, such asspeed, braking, steering or steering variation, temperature/overheating,etc., which might suggest or indicate erratic or improper driving thatcould be associated with drunk driving. Patterns could also be mined andutilized, such as may include such parameters over time, such aspatterns of high speed, erratic driving, or over-frequent braking, forexample. Day of the week, holiday, and time of day, or other generalenvironmental factors, could also be utilized. In some embodiments,sensors on the person of the driver, or a device such as a mobile deviceof the driver, could also be utilized, including location or evenphysiological parameters, or position, movement, or movement patterns ofdriver. Location of the vehicle, as determined from GPS, could alsofactor in, such as if the driver was or is driving in an area wheredrunk driving often occurs, or near a bar, for example.

Event definitions and thresholds can be expressed in terms of variousmultimodal sensor data. Events can include for example, a determinationthat drunk driving is occurring, a determination that it is occurringwith a certain degree of confidence or probability, etc.

Somewhat analogously, multimodal sensor data to include much of thatdescribed in connection with drunk driving detection could beincorporated into consumer and shopping related event detection, such asdetecting particular shopping activities, or shoplifting. This couldinclude, for example, the use of camera, 3D camera, depth camera, video,audio, light sensing at and around a particular shopping area, or even aparticular shelf or item, etc., as well as in connection with particularshoppers, including identity, characteristics, appearance, activity, andsurroundings.

Some embodiments of the invention relate to medical, hospital or patientmonitoring, or environmental or other monitoring. In some embodiments,external as well as wearable, internal or other sensors can be utilizedto sense various parameters, such as pulse rate or variability, heartrate or variability, respiratory rate or variability, pulse rate orvariability, body fluid analyte content, movement, nervous systemparameters, and many others. Such monitoring can include or beassociated with various monitoring, such as for monitoring any ofvarious patient parameters, organs, aspects, conditions, circumstances,or environmental or safety-related conditions or circumstances etc.These can include, for example, heart or heart-related monitoring ormanagement devices, such as a pacemaker, smart watch, smart belt, smartshirt, camera, or various other wearable monitoring or managementdevices. Furthermore, according to some embodiments, monitoring devicescan include, for example, devices to monitor physiological orenvironmental indications, evidence, signs or potential signs ofparticular conditions or dangerous or distress type situations. For apatient, person or animal, for example, such conditions can include, forexample, stroke, heart attack, etc. Furthermore, monitoring devices canbe used to monitor environmental or other conditions (whether or notassociated with a person or biological entity), such as for older orinfirm persons, such as might indicate a slip or fall, dangerousposition or location, etc.

In some embodiments, various of the above could be used in addition tovarious other parameters, many of which are generally described inconnection with drunk driving and consumer related application,including camera, 3D camera, depth camera, video, audio, light, patientidentity, appearance and position, patient surroundings, etc. Feed fromother medical monitoring systems associated with the patient could alsobe incorporated and utilized. Many different types of events could bedetermined, or determined with some degree of confidence or probability.These could include particular physiological events, such as occurrenceor status indicating a cardiac event or danger status, and many others.Definitions and thresholds relating to events can be expressed in termsof various multimodal sensor data.

Furthermore, in some embodiments, any of various monitoring and otherdevices can be used in connection with monitoring, and detected eventsthat may not be, or may not only be, associated with a person, such asevents associated with animals, plants, weather, environmentalconditions or occurrences, structures, objects, etc. For example,monitoring devices could be used to detect a weather or environmentaloccurrence such as a tree falling due to a storm or wind storm. Inaddition to monitoring in connection with the tree (or other object orenvironmental or structural feature, whether or not associated with aliving or biological entity), various parameters could be sensed andutilized, such as, in this example, storm parameters such as wind, rain,lightning, temperature, humidity, barometric pressure, etc. Such anapproach could be used in connection with various other events inconnection with living and non-living entities, including environmentalfeatures, structural features, etc.

Methods according to embodiments of the invention can apply to vehiclesof various types, including cars, planes, ships, trains etc., and atvarious levels in terms of components and component sets, andfine-grained, multimodal parameter measurements associated therewith. Ineach case, a MIB or portion thereof can be organized and structured tomatch the particular scenario, including the mapping and definitionstherein.

Many different contexts may also be represented. For example, in aconsumer context, multiple audio and video parameter data may be used indefinitions of events such as fraud, shoplifting, or others. In suchscenarios, a MIB or portion thereof could be structured and organized toreflect or model human behaviors, or behavioral events, for example. Ina medical or pain management scenario, parameters associated withmultiple monitored vital or other bodily system parameters, such aspulse rate, blood pressure, etc., may form part of a definition of amedical event, such as a heart attack, or an at-risk status, etc. Thepatient, or patient parameters, in such a scenario, could conceptuallybe represented by a MIB or a portion thereof. In a driving scenario,audio, video and other parameter data could be used, for example, indetermining that drunk driving is occurring or may be likely to beoccurring, etc. MIBs can be used in connection with many industries anddevices in those industries, such as manufacturing devices andfacilities. The MIB can be structured advantageously to reflect suchsets of parameters and associations with context-relevant events.

In some embodiments, in various scenarios, event data can be generatedor updated, and, after being detected within the network, actions may becaused, triggered or taken based at least in part on the operational orbehavioral event signaled. For example, in a battery overheatingscenario, signaling may be provided over the network to sendinstructions or commands to modify operation of the battery or itscomponents to prevent or mitigate damage or failure, or an alert can betriggered to the driver or some other entity, etc. In a consumer,retail, banking or other scenario, triggered actions could includesystem or human alerting, police or official entity alerting, alarms,enhanced monitoring, automatically locking or securing of devices orpremises, etc. In a driver scenario, action could include user alerting,police alerting, automatic shut-off, etc. In a patient scenario,triggered actions could include medical alerts, or even automaticadministration of medical or medicinal measures, etc.

In some embodiments, full user interface dashboards can be provided,which may include alerts or status information on various detectedevents and may allow action to be taken in response.

In some embodiments, logs can be created over time of parametermonitoring and measurements, such as using blockchains. Such logs canthen be used in many ways, such as, for example, as training data tomachine learning models, to optimize event database structure andmapping to events, etc.

Consider a use case where the disclosed techniques are leveraged tomonitor behaviors of shoppers within a supermarket. Behaviors in asupermarket can be defined according to location-specific orcontext-specific ontology. For example, the behaviors could be definedas having a hierarchy where the root of the hierarchy represents thecontext-specific setting, the supermarket in this case. A second levelin the hierarchy could include nodes representing activities associatedwith shopping or working to cover behaviors associated with consumersand employees respectively. Each of these nodes can be further brokendown into finer levels of detail to cover more specific behaviors;browsing, inspecting a product, purchasing, checking out, or otheractivities of a shopper for example. Such an ontology can be considereda domain ontology associated with the supermarket and can be implementedin a machine readable format, possibly based on the OWL standard format.One should appreciate that the disclosed inventive subject matter isconsidered to include software based tools that enable users to createdomain ontologies or otherwise manage domain ontologies for use withidentifying behaviors.

Continuing with the example, an analysis engine can ingest a supermarketdomain ontology OWL file to instantiate a memory model ofcontext-specific behaviors according to the encoded ontology. Theanalysis engine can further assign nodes and/or edges in the ontologywith object identifiers associated with a corresponding MIB, thusenabling use of SNMP in this use case. More specifically, the objectidentifiers can be organized according to a hierarchical tree structureas desired to facilitate the mapping to the behavior domain ontology andto facilitate addressing each behavior individually using software tools(e.g., OpenNSM).

Further each behavior within the domain ontology can be assignedcriteria defining when the behavior is considered to have (or will)occur as a function of the observed sensor data. The criteria can bedefined according to the attributes, attribute-value pairs for example,associated with sensor data where the attributes can be derived based onthe raw sensor data or derived from the raw sensor data. Continuing withthe example of the supermarket, the sensors could include securitycameras, cell phone sensors (e.g., camera, location, microphone, etc.),temperature sensors, or other types of sensors. As the security camerascapture digital representations of people in the supermarket, arecognition engine can apply one or more image processing techniques(e.g., SIFT, SURF, DAISY, etc.) to recognize people as employees (i.e.,recognized person) or shoppers (i.e., unrecognized person). Further,cell phone running a location-based app or employee name tags with RFIDcan be used to determine locations of persons within the supermarket.Such sensor information gives rise to a tuple of attributes, possibly inthe form of attribute-value pairs. When the tuple satisfies the criteriaof a behavior in the ontology, the MIB for the behavior can be updated,possibly triggering an SNPM message to another computer system. Thisapproach is considered advantageous because it leverages existingnetworking software tools (e.g., SNMP, OpenNMS, etc.) to map a sensedenvironment to behavior and generate notifications using trustedtechniques.

Yet another aspect of the shopping example includes using the behavioridentification as outlined above to planogram management. In view thatproducts are placed on fixtures according to planograms, interactions orother behaviors associated with the products can be used as evidence forthe effective of such planograms. For example, if a recognized person isa shopper, the shopper's behaviors such as browsing, price checking, orpurchasing can be mapped directly the planograms indicating, possibly asa heat map, the efficiency of the planogram or suggesting changes to theplanogram. Further, if the recognized person is an employee, theemployee's behaviors such as restocking, cleaning, or altering productplacement can be used to indicate that the supermarket is properlyleveraging a brand's planogram. Example techniques associated withvirtual planograms that could be adapted for use with the disclosedtechniques can be found in co-owned U.S. Pat. No. 9,430,752 toSoon-Shiong titled “Virtual Planogram Management, Systems, and Methods”filed Nov. 2, 2012. Although the previous example focused on the domainof shopping in a supermarket, it should be appreciated that thefundamental foundations (e.g., domain behavior ontology, MIB mapping,sensor to behavior, etc.) can be applied to other markets as wellincluding sports, healthcare management, hospitals, eSports, military,construction, physical therapy, janitorial services, educationalservices, or other markets.

In a more traditional networking setting, the disclosed inventivesubject matter can be leveraged to offer fine grained management ofbehaviors in a home automation IoT ecosystem. Just as in the supermarketcase, an ensemble of IoT devices can be treated as sensors whoseinformation combine to outline possible behaviors of the ecosystem. Forexample, a home can be instrumented with controllers and/or sensors forfans, lights, doorbells, pressure plates (e.g., occupancy sensor), infraread sensors, CO₂ sensors, electrical plugs, TVs, thermostats, cameras,appliances, or other devices. As the devices sense their environment,the sensor data can be analyzed to construct behaviors by a devicemanager (e.g., a set top box, Amazon® Echo®, home computer, etc.) andthen mapped to behaviors of the environment itself or of individualswithin the environment. Detected behaviors can trigger actions withinthe ecosystem; possibly generate an alarm when an illegal behavior hasbe identified based on observing a window seal breaking and anunrecognized person entering the home.

An astute reader will appreciate that due to the indirect mapping ofobserved sensor data to behaviors via ontologies one can also use thesensor data to map to more esoteric, yet useful concepts. Consider adata structure that represents a unit of electrical energy, say aKilowatt-hour. The data structure can be used to track the unit ofenergy from creation to use. For example, the data structure can beinstantiated when the unit of energy is created at the site of a solarcell on the roof of a house. Further the unit of energy can be stored ina battery. While stored, the data structure can include informationrelating to the physical nature of the battery (e.g., location,capacity, timestamp of storage, etc.). The data structure can be furtherupdated if the unit of energy is transferred to a different storagefacility, battery, or other location. Yet further the data structure canbe updated when the unit of energy is actually consumed. It isappreciated that this description represents a virtual management of aunit of energy. However, the data structure provides a way to trackenergy, similar to carbon credits, from creation to consumption.Further, with respect to the inventive subject matter, as the unit ofenergy flows through an energy management ecosystem as tracked by thedata structure, the unit of energy can be considered to interact withthe various aspects of the ecosystem. Such interactions can beconsidered behaviors that, again, can be mapped to an otology andmanaged as disclosed.

Not all embodiments will have concise or concrete definitions of abehavior with respect sensor data. In such scenarios, machine learningtechniques can be leveraged to map to the behaviors. As sensor data iscollected and correlated with behaviors, a training data set can becompiled that represents the sensor data (i.e., the collection ofattribute-value pairs, tuples, etc.) as categorized or classified as abehavior. If the sensor data falls within well-defined input signals,techniques such as support vector machines can be used to classify thebehaviors. If the sensor data is less well-defined with respect to theinput signals, neural networks can be used to classify the behaviors.Suitable machine learning software packages include scikit-learn (seeURL www.scikit-learn.org) or TensorFlow (see URL www.tensorflow.com).

Some embodiments of the present invention use or adapt techniquesdescribed in U.S. Application, publication number US20160117596A1(hereinafter, “US20160117596A1”), entitled “Reasoning Engines”, which ishereby incorporated herein by reference in its entirety. For example,techniques associated with correlations between aspects of anenvironment, as well as in object recognition, can be used in eventdatabase mapping structures and definitions in embodiments of theinvention. Still further, in some embodiments, such techniques, andreasoning engine output, can be used as training data to update machinelearning models and event databases. Furthermore, aspects of reasoningengines can be included in network management engines of embodiments ofthe invention.

Furthermore, types of sensors and interfaces as described inUS20160117596A1 can be advantageously used in embodiments of theinvention and can form aspects of multimodal sensor data. For example,characteristics or parameters that can be observed from sensors caninclude physical characteristics (e.g., size, shape, dimension, mass,weight, material, etc.), electrical characteristics (e.g., resistance,current, voltage, capacitance, reactance, inductance, permeability,permittivity, etc.), mechanical characteristics (e.g., stress, strain,torque, torsion, force, tension, compression, density, hardness, shear,friction, etc.), acoustical characteristics (e.g., spectrum, frequency,wavelength, absorption, amplitude, phase, etc.), chemicalcharacteristics (e.g., reactivity, pH, surface tension, surface energy,etc.), optical characteristics (e.g., absorptivity, color, luminosity,photosensitivity, reflectivity, refractive index, transmittance,spectrum, spectrum, frequency, wavelength, absorption, amplitude, phase,polarization, etc.), or other types of properties. Such properties canbe obtained directly from sensors directly or indirectly, and reflectchanges in aspects of environment, a patient's body or a building, forexample. Sensors and other items within a network or ecosystem cancommunicate among each other or remotely as desired. Elements within anecosystem may communicate over one or more wireless protocols, dependingon the nature of the element. For example, high bandwidth protocols maypossibly include Ultra-Wide Band (UWB), Wi-Fi, WiGIG, cellular networks(e.g., CDMA, GSM, etc.), or other types of broadband communications.Smaller elements can potentially use lower bandwidth protocols includingZigbee, wireless USB, Bluetooth, near field communications, or otherprotocols. The protocols used for exchanging data among elements ofecosystem or remotely can be configured to fit the bandwidth needs ofthe communication channel.

Some embodiments of the invention use or adapt techniques described inU.S. Pat. No. 8,504,740, entitled, “MILARRS Systems and Methods”, whichis hereby incorporated herein by reference in its entirety. For example,interfaces and configurations of interfaces as described therein can beused advantageously in embodiments of the present invention, forexample, in elements of instrumenting of devices, or in causingmodification of operation in response to event data detection, forinstance. For example, in some embodiments of the current invention, anyof currently known serial interfaces are contemplated, as well as thosethat are developed in the future. Currently completed standards for suchinterfaces include, for example, I2C, SPI, CAN, Profibus, RS 232, RS485, RS 422, USB, Ethernet, or even Gigabit Ethernet. Certainly, where agiven module has multiple serial interfaces, those interfaces can be ofdifferent types (i.e., operate using different standards). Depending onthe application, a contemplated serial interface could be built based ona set of general purpose programmable I/O (GPIO) pins. Such pins can bedriven by software running on a remote management module's (RMM's) CPUor the device's CPU implying the GPIO pins also provide a serial I/Ointerface. Connection from serial port 11 to the RMM 20 can beaccomplished in any desired manner, including hardwiring through phonelines, cables, connectors, soldering, or over the Internet. Typicalmanagement environment interfaces 27 are Ethernet, 802.11a/b/g, ATM, orothers. These interfaces are used to send data over packet switchednetworks including LANs, WANs, WLANs, or Internet.

Some embodiments of the invention use or adapt techniques described inU.S. Pat. No. 9,547,678, entitled, “Activity Recognition Systems andMethods”, which is hereby incorporated herein by reference in itsentirety. For example, techniques relating to database configuration aswell as recognition determination modelling, such as using graphing andscoring, can be used in embodiments of the present invention, such as instructuring of event databases, as well as in determining potentiallycomplex or functionally specified definitions of events.

Some embodiments of the invention use or adapt techniques described inU.S. Pat. No. 8,683,591, entitled, “Vector-Based Anomaly Detection”,which is hereby incorporated herein by reference in its entirety, whichincludes techniques for detecting anomalous behaviors associated withmultimode networks. The patent describes, among other things, techniquesin which vector-represented baselines are utilized and are a function ofseveral behavior metrics. Anomalies are then determined based onvariation from the baseline. An analogy can be drawn in some ways withsome embodiments of the present invention, in which events (which caninclude occurrence or continued occurrence of states, conditions orstatuses) are determined based on multiple sensed parameters,potentially in complex or function-specified ways. As such, vector-basedtechniques as described in U.S. Pat. No. 8,683,591, which may take intoaccount multiple parameters and may effectively define anomalies viavariation from a vector-based baseline, can be adapted for use invector-based, multimodal sensor data definitions of operational orbehavioral events, whether or not anomalies.

Some embodiments of the present invention use or adapt techniquesdescribed in U.S. Application, publication number US20150262036A1(hereinafter, “US20150262036A1”), entitled “Global Visual Vocabulary,Systems and Methods”, which is hereby incorporated herein by referencein its entirety. US20150262036A1 includes description of techniques inwhich a hierarchically structured and indexed global vocabulary anddescriptor space, which can be used in populated a database, and canprovide information on recognized images or objects. In some embodimentsof the present invention, elements of structuring and organizationdescribed in US20150262036A1 can be adapted and used in structuring andorganization of event databases, such as MIBs, that contain hierarchicalor ontologically structured representations of operational or behavioralevents.

FIG. 9 illustrates components of one embodiment of an environment inwhich the invention may be practiced. Not all of the components may berequired to practice the invention, and variations in the arrangementand type of the components may be made without departing from the spiritor scope of the invention. As shown, the system 900 includes a network910, which may broadly include one or more networks, such as cloudnetworks, wired or wireless networks, local area networks (“LANs”)/widearea networks (“WANs”). The system 900 further includes one or morewired or wireless devices 902-906, such as may include or be coupledwith one or more servers 907, such as may include, or include elementsof, for example, network management engine 902, and one or more datastores or databases 808, such as Event database 904. Various of theclient devices 902-906 may include a wide array of types of devices,which may or may not be incorporated or associated with large or otherdevices, including, for example, desktop computers, laptop computers,set top boxes, tablets, cell phones, smart phones, batteries, vehiclebatteries, power management devices, network management devices,monitoring devices, medical or health care related devices, video oraudio devices, tracking devices, and many others. Each of the devices902-906 may include various elements or components including wired orwireless interfaces, software or hardware instrumentation, etc.

FIG. 10 illustrates a block diagram of an electronic device 1000 thatcan implement one or more aspects of a network management system ormethod according to one embodiment of the invention. Instances of theelectronic device 1000 may include servers, e.g. servers 907-908, anddevices, e.g. devices 902-906. In general, the electronic device 1000can include a processor 1002, memory 1030, a power supply 1006, andinput/output (I/O) components 1040, e.g., microphones, speakers,displays, touchscreens, keyboards, keypads, GPS components, etc., whichmay be operable, for example, to provide graphical user interfaces. Theelectronic device 1000 can also include a communications bus 1004 thatconnects the aforementioned elements of the electronic device 1000.Network interfaces 1014 can include a receiver and a transmitter (ortransceiver), and an antenna for wireless communications.

The processor 1002 can include one or more of any type of processingdevice, e.g., a central processing unit (CPU). Also, for example, theprocessor can be central processing logic. Central processing logic, orother logic, may include hardware, firmware, software, or combinationsthereof, to perform one or more functions or actions, or to cause one ormore functions or actions from one or more other components. Also, basedon a desired application or need, central processing logic, or otherlogic, may include, for example, a software controlled microprocessor,discrete logic, e.g., an application specific integrated circuit (ASIC),a programmable/programmed logic device, memory device containinginstructions, etc., or combinatorial logic embodied in hardware.Furthermore, logic may also be fully embodied as software.

The memory 1030, which can include RAM 1012 and ROM 1032, can be enabledby one or more of any type of memory device, e.g., a primary (directlyaccessible by the CPU) or secondary (indirectly accessible by the CPU)storage device (e.g., flash memory, magnetic disk, optical disk). TheRAM can include an operating system 1021, data storage 1024, which mayinclude one or more databases, and programs and/or applications 1022,which can include, for example, software aspects of the networkmanagement program 103. The ROM 1032 can also include BIOS 1020 of theelectronic device.

The Network Management Program 1023 is intended to broadly include orrepresent all programming, applications, algorithms, software and othertools necessary to implement or facilitate methods and systems accordingto embodiments of the invention, which may include elements of a networkmanagement engine, such as network management engine 102. The elementsof the Network Management Program 1023, or of a network managementengine, may exist on a single server computer or be distributed amongmultiple computers, devices or entities.

The power supply 1006 contains one or more power components andfacilitates supply and management of power to the electronic device1000.

The input/output components, including I/O interfaces 1040, can include,for example, any interfaces for facilitating communication between anycomponents of the electronic device 1000, components of external devices(e.g., components of other devices of the network or system 900), andend users. For example, such components can include a network card thatmay be an integration of a receiver, a transmitter, and one or moreinput/output interfaces. A network care, for example, can facilitatewired or wireless communication with other devices of a network. Incases of wireless communication, an antenna can facilitate suchcommunication. Also, some of the input/output interfaces 1040 and thebus 1004 can facilitate communication between components of theelectronic device 1000, and in an example can ease processing performedby the processor 1002. In some embodiments, the device 1000 may includeor be coupled to network management device instrumentation, such asnetwork management related device instrumentation 110.

Where the electronic device 1000 is a server, it can include a computingdevice that can be capable of sending or receiving signals, e.g., via awired or wireless network, or may be capable of processing or storingsignals, e.g., in memory as physical memory states. The server may be anapplication server that includes a configuration to provide one or moreapplications via a network to another device. Also, an applicationserver may, for example, host a Web site that can provide a userinterface for administration of example aspects of the NetworkManagement Program 1023.

Any device capable of sending, receiving, and processing data over awired and/or a wireless network may act as a server, such as infacilitating aspects of implementations of the Network ManagementProgram 1023. Thus, devices acting as a server may include devices suchas dedicated rack-mounted servers, desktop computers, laptop computers,set top boxes, integrated devices combining one or more of the precedingdevices, other devices, etc.

Servers may vary in widely in configuration and capabilities, but theygenerally include one or more central processing units, memory, massdata storage, a power supply, wired or wireless network interfaces,input/output interfaces, and an operating system such as Windows Server,Mac OS X, Unix, Linux, FreeBSD, etc.

A server may include, for example, a device that is configured, orincludes a configuration, to provide data or content via one or morenetworks to another device, such as in facilitating aspects of anexample Network Management Program 1023. One or more servers may, forexample, be used in hosting a Web site.

Servers may also, for example, provide a variety of services, such asWeb services, third-party services, audio services, video services,email services, instant messaging (IM) services, SMS services, MMSservices, FTP services, voice or IP (VOIP) services, calendaringservices, phone services, advertising services etc., all of which maywork in conjunction with example aspects of an example NetworkManagement Program 923. Content may include, for example, text, images,audio, video, advertisements, etc.

In example aspects of the Network Management Program 1023, devices mayinclude, for example, any device capable of sending and receiving dataover a wired and/or a wireless network. Such devices may include desktopcomputers as well as portable devices such as cellular telephones, smartphones, display pagers, radio frequency (RF) devices, infrared (IR)devices, handheld computers, tablets, GPS-enabled devices tabletcomputers, sensor-equipped devices, laptop computers, set top boxes,monitoring devices, medically related devices, consumer related devices,tracking devices, battery or power management devices, and many others.Devices may range widely in terms of capabilities and features. Forexample, a cell phone, smart phone or tablet may have a numeric keypadand a few lines of monochrome LCD display on which only text may bedisplayed. In another example, a Web-enabled client device may have aphysical or virtual keyboard, data storage (such as flash memory or SDcards), accelerometers, gyroscopes, GPS or other location-awarecapability, and a 2D or 3D touch-sensitive color screen on which bothtext and graphics may be displayed.

Devices, such as client devices 902-906, for example, may run a varietyof operating systems, including personal computer operating systems suchas Windows, iOS or Linux, and mobile operating systems such as iOS,Android, and Windows Mobile, etc. Client devices may be used to run oneor more applications that are configured to send or receive data fromanother computing device. Client applications may provide and receivetextual content, multimedia information, etc. Client applications mayperform actions such as browsing webpages, using a web search engine,sending and receiving messages via email, SMS, or MMS, playing games(such as fantasy sports leagues), receiving advertising, watchinglocally stored or streamed video, or participating in social networks.

In example aspects of the Network Management Program 1023, one or morenetworks, such as networks 1010 or 1012, for example, may couple, suchas via interfaces or instrumentation, servers and devices with othercomputing devices, including through wireless network to client devices.A network may be enabled to employ any form of computer readable mediafor communicating information from one electronic device to another. Anetwork may include the Internet in addition to local area networks(LANs), wide area networks (WANs), direct connections, such as through auniversal serial bus (USB) port, other forms of computer-readable media,or any combination thereof. On an interconnected set of LANs, includingthose based on differing architectures and protocols, a router acts as alink between LANs, enabling data to be sent from one to another.

Communication links within LANs may include twisted wire pair or coaxialcable, while communication links between networks may utilize analogtelephone lines, cable lines, optical lines, full or fractionaldedicated digital lines including T1, T2, T3, and T4, IntegratedServices Digital Networks (ISDNs), Digital Subscriber Lines (DSLs),wireless links including satellite links, or other communications linksknown to those skilled in the art. Furthermore, remote computers andother related electronic devices could be remotely connected to eitherLANs or WANs via a modem and a telephone link.

A wireless network, such as wireless network 1010, as in an exampleNetwork Management Program 1023, may couple devices with a network. Awireless network may employ stand-alone ad-hoc networks, mesh networks,Wireless LAN (WLAN) networks, cellular networks, etc.

A wireless network may further include an autonomous system ofterminals, gateways, routers, or the like connected by wireless radiolinks, or the like. These connectors may be configured to move freelyand randomly and organize themselves arbitrarily, such that the topologyof wireless network may change rapidly. A wireless network may furtheremploy a plurality of access technologies including 2nd (2G), 3rd (3G),4th (4G) generation, Long Term Evolution (LTE) radio access for cellularsystems, WLAN, Wireless Router (WR) mesh, etc. Access technologies suchas 2G, 2.5G, 3G, 4G, 5G and future access networks may enable wide areacoverage for client devices, such as client devices with various degreesof mobility. For example, wireless network may enable a radio connectionthrough a radio network access technology such as Global System forMobile communication (GSM), Universal Mobile Telecommunications System(UMTS), General Packet Radio Services (GPRS), Enhanced Data GSMEnvironment (EDGE), 3GPP Long Term Evolution (LTE), LTE Advanced,Wideband Code Division Multiple Access (WCDMA), Bluetooth, 802.11b/g/n,etc. A wireless network may include virtually any wireless communicationmechanism by which information may travel between client devices andanother computing device, network, etc.

Internet Protocol may be used for transmitting data communicationpackets over a network of participating digital communication networks,and may include protocols such as TCP/IP, UDP, DECnet, NetBEUI, IPX,Appletalk, and the like. Versions of the Internet Protocol include IPv4and IPv6. The Internet includes local area networks (LANs), wide areanetworks (WANs), wireless networks, and long haul public networks thatmay allow packets to be communicated between the local area networks.The packets may be transmitted between nodes in the network to siteseach of which has a unique local network address. A data communicationpacket may be sent through the Internet from a user site via an accessnode connected to the Internet. The packet may be forwarded through thenetwork nodes to any target site connected to the network provided thatthe site address of the target site is included in a header of thepacket. Each packet communicated over the Internet may be routed via apath determined by gateways and servers that switch the packet accordingto the target address and the availability of a network path to connectto the target site.

A “content delivery network” or “content distribution network” (CDN), asmay be used in an example Network Management Program 1023, generallyrefers to a distributed computer system that comprises a collection ofautonomous computers linked by a network or networks, together with thesoftware, systems, protocols and techniques designed to facilitatevarious services, such as the storage, caching, or transmission ofcontent, streaming media and applications on behalf of contentproviders. Such services may make use of ancillary technologiesincluding, but not limited to, “cloud computing,” distributed storage,DNS request handling, provisioning, data monitoring and reporting,content targeting, personalization, and business intelligence. A CDN mayalso enable an entity to operate and/or manage a third party's Web siteinfrastructure, in whole or in part, on the third party's behalf.

A peer-to-peer (or P2P) computer network relies primarily on thecomputing power and bandwidth of the participants in the network ratherthan concentrating it in a given set of dedicated servers. P2P networksare typically used for connecting nodes via largely ad hoc connections.A pure peer-to-peer network does not have a notion of clients orservers, but only equal peer nodes that simultaneously function as both“clients” and “servers” to the other nodes on the network.

The various embodiments are described with reference to the accompanyingdrawings, which form a part hereof, and which show, by way ofillustration, specific examples of practicing the embodiments. Thisspecification may, however, be embodied in many different forms andshould not be construed as limited to the embodiments set forth herein;rather, these embodiments are provided so that this specification willbe thorough and complete, and will fully convey the scope of theinvention to those skilled in the art. Among other things, thisspecification may be embodied as methods or devices. Accordingly, any ofthe various embodiments herein may take the form of an entirely hardwareembodiment, an entirely software embodiment or an embodiment combiningsoftware and hardware aspects. The following specification is,therefore, not to be taken in a limiting sense.

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The phrase “in one embodiment” as used herein doesnot necessarily refer to the same embodiment, though it may.Furthermore, the phrase “in another embodiment” as used herein does notnecessarily refer to a different embodiment, although it may. Thus, asdescribed below, various embodiments of the invention may be readilycombined, without departing from the scope or spirit of the invention.

In addition, as used herein, the term “or” is an inclusive “or”operator, and is equivalent to the term “and/or,” unless the contextclearly dictates otherwise. The term “based on” is not exclusive andallows for being based on additional factors not described, unless thecontext clearly dictates otherwise. In addition, throughout thespecification, the meaning of “a,” “an,” and “the” includes pluralreferences. The meaning of “in” includes “in” and “on.”

It is noted that description is not intended as an extensive overview,and as such, concepts may be simplified in the interests of clarity andbrevity.

The specification is to be understood as being in every respectillustrative and exemplary, but not restrictive, and the scope of theinvention disclosed herein is not to be determined from thespecification, but rather from the claims as interpreted according tothe full breadth permitted by the patent laws. It is to be understoodthat the embodiments shown and described herein are only illustrative ofthe principles of the present invention and that various modificationsmay be implemented by those skilled in the art without departing fromthe scope and spirit of the invention. Those skilled in the art couldimplement various other feature combinations without departing from thescope and spirit of the invention.

1-40. (canceled)
 41. A network management system, comprising: one or more processors; an event database providing a mapping of multimodal sensor data to events of one or more devices coupled to a network, wherein the event database comprises a Management Information Base (MIB), and wherein the MIB is organized relative to operational or behavioral events; and a memory storing software instructions that, when executed by the one or more processors, cause the one or more processors to: obtain a set of multimodal sensor data relating to a device of the one or more devices; train one or more machine learning models based at least on a representation of the obtained set of multimodal sensor data; update the mapping based on the one or more trained machine learning models; determine, utilizing the updated mapping, a match of the obtained set of multimodal sensor data to a first event associated with the device; and based at least in part on the determined match, cause generation or updating of first event data associated with the first event, the first event data being accessible by network management software of the network management system.
 42. The system of claim 41, wherein the events of the one or more devices comprise operational events and the first event is related to at least one of chemical, mechanical and electrical operation of the device of the one or more devices.
 43. The system of claim 41, wherein an event of the events of the one or more devices is associated with a biological, physiological, electrical, magnetic, electromagnetic, environmental, structural, or weather-related occurrence, condition or phenomenon.
 44. The system of claim 41, wherein the events of the one or more devices comprise behavioral events and the first event is related to behavior of an entity interacting with the device.
 45. The system of claim 41, wherein the first event data reflects occurrence, continued occurrence or repeated occurrence of the first event.
 46. The system of claim 41, wherein the multimodal sensor data comprises multiple outputs or measurements, each relating to a different sensed parameter.
 47. The system of claim 41, wherein the one or more processors are further caused to: correlate the multimodal sensor data with one or more behaviors; and generate a training data set to represent the multimodal sensor data.
 48. The system of claim 47, wherein the training data set represents the multimodal sensor data classified as a behavior.
 49. The system of claim 48, wherein the classification of the multimodal sensor data is based on a classification by support vector machines.
 50. The system of claim 48, wherein the classification of the multimodal sensor data is based on a classification by one or more neural networks.
 51. The system of claim 41, wherein the one or more processors are further caused to: iteratively perform at least one of: obtaining one or more additional sets of multimodal sensor data, training the one or more machine learning models based at least on a representation of the obtained one or more additional sets of multimodal sensor data; and updating the event database based on the iteratively trained one or more machine learning models.
 52. The system of claim 41, wherein the multimodal sensor data is collected, and computationally modified, or filtered, at the device of the one or more devices.
 53. The system of claim 41, wherein the multimodal sensor data is collected, and computationally processed or filtered, at the device of the one or more devices prior to wireless transmission over the Internet and prior to storage in the event database.
 54. The system of claim 41, wherein the match is determined based on the multimodal sensor data falling within a range or being above a threshold.
 55. The system of claim 41, wherein: the match is determined based on the multimodal sensor data falling within a range or being above a threshold; and the range or the threshold requires that, for each of multiple single outputs or single measurements of the multimodal sensor data, the single output or the single measurement falls within a specified range or is above a specified threshold, the specified range or the specified threshold relating to the single output or the single measurement.
 56. The system of claim 41, wherein: the multimodal sensor data comprises multiple sensor outputs, each relating to a different sensed parameter; and only a combination of all of the multiple sensor outputs, and not any single one of the multiple sensor outputs or any subset of the multiple sensor outputs, matches to the first event.
 57. The system of claim 41, wherein the one or more processors are further caused to: cause monitoring for, and detection of, the first event data utilizing the network management software.
 58. The system of claim 41, wherein the one or more processors are further caused to: upon detection of the first event data by the network management software, cause generation of an alert.
 59. The system of claim 41, wherein the one or more processors are further caused to: upon detection of the first event data by the network management software, based at least in part on the on the first event, cause modification of function of the device to better optimize function of the device.
 60. The system of claim 41, wherein the event database specifies, in terms of multimodal sensor data, definitions or thresholds relating to events.
 61. The system of claim 41, wherein an event of the events of the one or more devices relates to operation or behavior of a device or of a person.
 62. The system of claim 41, wherein an event of the events of the one or more devices is associated with a person, animal, plant, or organ.
 63. The system of claim 41, wherein an event of the events of the one or more devices is associated with a feature, aspect, element, portion, or condition of a living entity.
 64. The system of claim 41, wherein the one or more processors are further caused to: utilizing event data, determine and store data regarding sensed operation or behavior; track and store data regarding outcomes resulting from sensed operation or behavior; and utilize the one or more machine learning models, determine patterns associating sensed operation or behavior with outcomes.
 65. The system of claim 41, wherein the one or more processors are further caused to: store a log over time of varying multimodal sensor data associated with the device of the one or more devices.
 66. The system of claim 65, wherein the one or more processors are further caused to train one or more machine learning models based on at least a part of the stored log over time of varying multimodal sensor data.
 67. The system of claim 41, wherein the one or more processors are further caused to: store a log over time of varying multimodal sensor data associated with the device of the one or more devices using one or more blockchains.
 68. The system of claim 41, wherein the device comprises a battery or power cell and wherein the multimodal sensor data comprises data relating to each of multiple parameters of the battery or power cell.
 69. The system of claim 41, wherein the device comprises a component of a battery or power cell and wherein the multimodal sensor data comprises data relating to the component.
 70. The system of claim 41, wherein the device comprises a component of a vehicle.
 71. The system of claim 41, wherein the device comprises a medical device.
 72. The system of claim 41, wherein the first event is associated with a patient or a consumer.
 73. The system of claim 41, wherein the one or more processors are further caused to: generate a user interface dashboard displaying an indication or alert of the first event; and allowing a user, through the dashboard, to take action based on the first event.
 74. A network management method, comprising: maintaining an event database providing a mapping of multimodal sensor data to events of one or more devices coupled to a network, wherein the event database comprises a Management Information Base (MIB), and wherein the MIB is organized relative to operational or behavioral events; obtaining a set of multimodal sensor data relating to a device of the one or more devices; training one or more machine learning models based at least on a representation of the obtained set of multimodal sensor data; updating the mapping based on the one or more trained machine learning models; determining, utilizing the updated mapping, a match of the obtained set of multimodal sensor data to a first event associated with the device; and based at least in part on the determined match, causing generation or updating of first event data associated with the first event, the first event data being accessible by network management software of the network management system; detecting the first event data; and based at least in part on the detection the first event data, causing a modification of operation of the device.
 75. The method of claim 74, wherein causing the modification of the operation of the device is to better optimize operation of the device based at least in part on the first event data.
 76. The method of claim 74, wherein the first event is related to at least one of chemical, mechanical or electrical operation of the device.
 77. A non-transitory computer readable medium or media containing instructions for executing a method comprising: maintaining an event database providing a mapping of multimodal sensor data to events of one or more devices coupled to a network, wherein the event database comprises a Management Information Base (MIB), and wherein the MIB is organized relative to operational or behavioral events; obtaining a set of multimodal sensor data relating to a device of the one or more devices; training one or more machine learning models based at least on a representation of the obtained set of multimodal sensor data; updating the mapping based on the one or more trained machine learning models; determining, utilizing the updated mapping, a match of the obtained set of multimodal sensor data to a first event associated with the device; based at least in part on the determined match, causing generation or updating of first event data associated with the first event, the first event data being accessible by network management software of the network management system; detecting the first event data; and based at least in part on the detection the first event data, causing a modification of operation of the device.
 78. The computer readable medium or media of claim 77, wherein the first event is related to at least one of chemical, mechanical or electrical operation of the device. 